Skip to content

[fix](variant) Forward-port Parquet and external Variant fixes to master - #66413

Open
Gabriel39 wants to merge 20 commits into
apache:masterfrom
Gabriel39:dev/forward-port-66397-66302-master
Open

[fix](variant) Forward-port Parquet and external Variant fixes to master#66413
Gabriel39 wants to merge 20 commits into
apache:masterfrom
Gabriel39:dev/forward-port-66397-66302-master

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Forward-port the Parquet nullable selection optimization from branch-4.1 to the latest master.

  • Forward-port Iceberg Variant reads to master's plugin-driven connector implementation.

  • Preserve File Scanner V2 safety checks, mixed-version backend gating, typed-leaf projection, and delete-only MERGE behavior.

  • Resolve the latest-master benchmark and deferred-column layout conflicts while retaining coverage from both sides.

  • Adapt the catalog Hadoop-property concurrency fix to master's storage adapter architecture by atomically publishing an immutable shared snapshot.

  • Expand Iceberg Variant reader coverage for primitive types, nested containers, multi-file scans, row groups, delete vectors, equality/position deletes, and lazy materialization.

  • Preserve append atomicity for nullable, STRUCT, ARRAY, and MAP destinations when lazy Variant fallback discovers corrupt input.

  • Cache Variant schema presence so ordinary Parquet scans avoid Variant-specific planning and statistics work.

  • Forward-port compound Parquet Page Index pruning, combining AND ranges by intersection and OR ranges by union while preserving residual evaluation.

  • Keep compound predicate pruning behind master's metadata-pruning safety fence.

  • Forward-port the hardened projected shredded Variant lifecycle fixes, preserving projected state across exchange, TopN, truncation, mixed file layouts, and scanner-profile destruction.

  • Add native Paimon reads for schema-matched unannotated Variant carriers without misclassifying ordinary structs.

  • Apply the external-table guardrail stabilization by removing nondeterministic scanner-distribution assertions while retaining deterministic parallel correctness coverage.

  • Fix late runtime-filter refreshes so predicate/non-predicate reclassification is accepted only when physical slots and projections remain unchanged.

  • Preserve Paimon nested timestamp semantics with Parquet's independent predicate and deferred-output projections after rebasing.

  • Map Paimon Variant schemas to the execution-only VariantV2 carrier so both JNI and native scan plans remain queryable in Nereids.

  • Align the Iceberg nested-path EXPLAIN assertion and Paimon six-column DESC golden with master's plugin-driven metadata format.

  • Forward-port Iceberg schema-evolution and nested partition-write hardening, including nested defaults and rollback-safe append behavior.

  • Preserve Doris floating-point equality semantics for Parquet pruning across NaN values and signed zero.

  • Safely localize nested Parquet Bloom probes while preserving predicate order, schema-validation barriers, and conservative fallback.

Original pull requests

Verification

  • FE reactor build, Checkstyle, and targeted tests: 277 tests passed for the Parquet and Iceberg forward ports.

  • CatalogPropertyTest: 2 tests passed for atomic publication and snapshot immutability.

  • Final FE Checkstyle passed for all 74 reactor modules after rebasing onto master at a82564ced5.

  • Final post-rebase FE reactor compilation and targeted tests passed: 104 tests across ConnectorPluginSurfaceTest, IcebergWritePlanProviderTest, RequestPropertyDeriverTest, and PluginDrivenTableSinkTest.

  • Repository clang-format 16 verification passed for all 85 affected C/C++ source and header files.

  • BE ASAN targeted tests: 408 tests from 29 suites passed for the earlier forward ports.

  • Focused BE ASAN compound Page Index tests: 2 tests from 2 suites passed.

  • Final focused BE ASAN verification after rebasing: 225 tests from 18 suites passed, including late runtime-filter refresh, nested timestamp semantics, Variant, Parquet, and Paimon coverage.

  • Clang-format 16 dry run passed for all 85 C/C++ source and header files changed by this PR.

  • PaimonTypeMappingReadTest: 3 tests passed with Maven build cache disabled.

  • ConnectorColumnConverterTest#testComputeVariantCarrierConversion: 1 test passed with Maven build cache disabled.

  • Full FE Checkstyle passed for all 74 reactor modules after the Paimon Variant mapping fix.

  • Regression framework build and tests passed: 3 tests, 0 failures; the Paimon DESC golden was also validated as six columns with an empty Extra field.

  • git diff --check passed for the Groovy source changes; the generated Paimon .out rows retain the required trailing delimiter for an empty Extra field.

  • Full FE Checkstyle passed for all 74 reactor modules after applying [fix](iceberg) Harden schema evolution and nested partition writes #66529.

  • Targeted FE Iceberg planner, metadata, manifest-cache, and statement-scope tests passed after applying [fix](iceberg) Harden schema evolution and nested partition writes #66529.

  • BE ASAN floating-point pruning verification passed: 58 tests from 6 suites after applying [fix](be) Preserve floating-point equality in Parquet pruning #66470.

  • Final BE ASAN nested-pruning verification passed: 356 tests from 6 suites after applying [fix](be) Safely prune nested Parquet columns with Bloom filters #66471 and the master API adaptations.

  • Clang-format 16 verification passed for all 47 C/C++ source and header files changed by [fix](iceberg) Harden schema evolution and nested partition writes #66529, [fix](be) Preserve floating-point equality in Parquet pruning #66470, and [fix](be) Safely prune nested Parquet columns with Bloom filters #66471.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39
Gabriel39 marked this pull request as ready for review August 4, 2026 04:04
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30875467070

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: four blocking issues remain.

  • Variant footer/page-index pruning can suppress an earlier error-preserving conjunct.
  • Late runtime-filter refresh can accept shifted deferred Variant output slots and index outside the active file block.
  • Delete-only Variant MERGE is not safe for new-FE/old-BE rolling upgrades.
  • One deterministic regression result bypasses the required generated golden file.

Checkpoint conclusions:

  • Goal and data correctness: The forward-port covers native Parquet Variant reads, nullable selection, planner/access-path plumbing, metadata COUNT, and delete-only MERGE, with broad unit/regression coverage; the two scan correctness defects above mean the goal is not yet safely achieved.
  • Scope and parallel paths: The change is cohesive but large. Footer/page pruning, eager/deferred projection, native/legacy scanner gates, and read/write paths were traced end to end. The mirrored page-index defect is covered by the first inline comment.
  • Concurrency and lifecycle: Catalog storage bindings and shredded-state ownership/COW were checked without another defect. Late request activation at row-group boundaries is not safe because deferred positions are not preserved (inline comment).
  • Compatibility and protocol: New Thrift plumbing defaults correctly for old-FE/new-BE, but new-FE/old-BE delete-only Variant MERGE lacks a query-wide capability fence (inline comment).
  • Tests and observability: The PR reports targeted FE, connector, BE ASAN, and format checks, and adds useful profiles. I did not rerun builds/tests because the authoritative review bundle forbids it. Missing coverage includes unsafe-conjunct metadata pruning, two-root late-RF refresh, mixed-version writer omission, and the signed-selector golden result.
  • Transactions/persistence/configuration: No new persistence or dynamic-configuration defect was found; delete-file lifecycle otherwise remains fenced and errors propagate.

User focus: review_focus.txt contains no additional guidance, so the entire PR was reviewed.

Review completion: Three rounds converged on this frozen four-comment set; all other candidates were either disproved by upstream invariants or dismissed with concrete code evidence.

const std::vector<std::unique_ptr<ParquetColumnSchema>>& file_schema,
const format::FileScanRequest& request, const cctz::time_zone* timezone) {
for (const auto& conjunct : request.conjuncts) {
const auto predicate = extract_variant_shredded_predicate(conjunct);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Fence Variant metadata pruning at the first unsafe conjunct

This loop can skip an earlier non-extractable, error-preserving conjunct and prune from a later Variant comparison before any rows are evaluated. For example, with assert_true(id != 0, 'boom') followed by CAST(v['n'] AS INT) > 3000, a row group containing id = 0 but with shredded n.max <= 3000 is discarded here, so the required error never occurs. TableReader::_is_safe_to_pre_execute() stops split pruning at this barrier, and build_predicate_conjunct_schedule() disables staged compaction for the same reason, but both this footer loop and the page-index loop at 1712 ignore it. Please carry the original safe-prefix boundary into the localized request (slotless barriers may otherwise disappear), apply it to both loops, and add an error-preservation differential test.

// Signed integer selectors are array indexes, even when a shredded object has a key with the
// same serialized token. The ambiguous scanner path must retain enough state for both results.
List<List<Object>> signedSelectorRows = sql """
SELECT CAST(v[-1] AS INT), CAST(v['-1'] AS INT)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Record this deterministic result through the golden file

This stable one-row result is currently hardcoded with assertEquals, so it bypasses the suite's generated .out contract. The repository testing rules require determined regression results to use qt/order_qt rather than assertion-style checks. Please make this a named golden query (with deterministic ordering if the fixture can grow) and regenerate test_iceberg_variant_read.out so the signed-selector behavior is reviewed and maintained like the neighboring cases.

file_request->local_positions = *fixed_local_positions;
}
file_request->non_predicate_positions.clear();
file_request->conjuncts.clear();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve deferred output slots across late-filter refresh

refresh_conjuncts() seeds only local_positions, while this clears and later reallocates non_predicate_positions. With two independent Variant roots, the initial layout can be v1 predicate/output at 0/1 and v2 at 2/3; rebuilding from retained locals {0,2} assigns the outputs to 3/4. same_physical_scan_layout() compares only the local map and each root's predicate projection, so it accepts that shifted request even though the active file-block template still has four slots. At the next row group the Parquet scheduler uses the refreshed output positions, overwrites slot 3 and then indexes slot 4 out of range. Please preserve the active deferred-position map and compare both deferred positions and output projection trees before queueing a refresh; add a multi-row-group late-RF test with two deferred roots.

const auto& merge_sink = _t_sink.iceberg_merge_sink;
// An old FE cannot produce delete-only plans, so an unset flag retains its data-writer path.
_writes_data_files = !merge_sink.__isset.writes_data_files || merge_sink.writes_data_files;
// Missing means an old FE plan, which predates SQL MERGE cardinality validation.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Fence writer omission for old BEs during rolling upgrade

A new FE can now allow a delete-only MERGE on a Variant table and send writes_data_files=false, but an old BE skips this unknown Thrift field and still constructs VIcebergTableWriter. Its init_properties() parses the full Iceberg schema_json, and the old parser has no variant primitive, so a fragment placed on that BE fails while the same fragment succeeds on a new BE. The adjacent cardinality capability is disabled through the query-wide execution version for exactly this rolling-upgrade reason; please add an equivalent capability fence here (or reject this plan in FE until all participating BEs support writer omission) and cover the mixed-version case.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 37.04% (30/81) 🎉
Increment coverage report
Complete coverage report

yiguolei
yiguolei previously approved these changes Aug 4, 2026
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Aug 4, 2026
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@Gabriel39
Gabriel39 force-pushed the dev/forward-port-66397-66302-master branch from 154db63 to 6adba50 Compare August 4, 2026 07:57
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot removed the approved Indicates a PR has been approved by one committer. label Aug 4, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: three additional P1 blockers remain. I deduplicated against the four existing live threads, which continue to cover unsafe Variant metadata-pruning order, late-filter deferred-slot refresh, the regression golden contract, and old-BE merge-writer omission.

Checkpoint conclusions:

  • Goal and proof: The PR aims to forward-port native Iceberg Variant reads/writes, physical-leaf pruning, and fused nullable selection. Broad unit and regression coverage is present, but the connector API, rolling-upgrade scan admission, and metadata-COUNT snapshot gaps mean the goal is not safely complete.
  • Scope and focus: The change is large but cohesive around external Variant execution. review_focus.txt adds no extra guidance, so the whole PR was reviewed.
  • Concurrency: No new thread is introduced. The private shredded-state materialization-cache mutex, its append/reset/read ordering, catalog publication, scanner scheduling, and late runtime-filter activation were traced; no new concurrency issue survived beyond the existing deferred-slot thread.
  • Lifecycle: Connector class loading, scan initialization, statement snapshot pinning, file-local projection, block reuse, and merge sink open/close were traced. The metadata-COUNT escape hatch is incorrectly decided before the handle used for planning is pinned (inline).
  • Configuration: No new configuration item or dynamic-reload contract is introduced.
  • Compatibility: Two public connector SPI methods were added without the required API-major/baseline update (inline). Read-side Variant admission also mistakes a cloud-only smooth-upgrade marker for a general old-BE capability fence (inline).
  • Parallel paths: Cloud and community upgrades, root and leaf projections, native and legacy scanner gates, metadata and real-range COUNT, and all merge clause shapes were compared. Delete-only merge propagation is complete for new participants; its old-BE failure remains covered by the existing live thread.
  • Conditional logic: The metadata-only COUNT and backend-marker conditions are not sufficient for the states they claim to prove (inline). Other new projection, fallback, and selection gates were checked against their upstream invariants.
  • Error handling and memory safety: Status/exception propagation, footer corruption checks, recursive column exclusivity, direct-leaf ownership, nullable alignment, and conversion-failure remapping were checked without another distinct defect.
  • Data correctness: Existing live threads cover unsafe metadata pruning and shifted scan coordinates. The new snapshot and mixed-version findings can also route unsupported Variant decoding and are blocking.
  • Tests: Coverage is broad, but it lacks connector-major enforcement for the reachable handle/provider surface, a non-cloud old-BE scan case, and a pinned snapshot whose COUNT summary must fall back to files. The existing live P2 covers the deterministic result that bypasses the generated golden file.
  • Test results: I did not run builds or tests because the authoritative review bundle requires a static-only review; reported PR results were therefore not independently verified.
  • Observability: New scan profiles cover the important reader paths, and no distinct logging or metrics blocker was found. The upgrade and snapshot mismatches need admission-time correctness rather than post-failure observability.
  • Transactions and persistence: No Doris EditLog or transaction-state change is introduced. Iceberg snapshot selection and write lifecycle were reviewed; snapshot consistency is the blocking read-side issue.
  • Writes and FE-BE variables: The new write flag survives planner clones and new-version sink lifecycles, and its old-FE/new-BE default is conservative. New-FE/old-BE writer omission remains the existing live blocker; the new read carrier has the separate ordinary-upgrade blocker inline.
  • Performance and other risks: Physical projection, page/footer pruning, allocation/COW, and fused nullable hot paths were examined. No additional substantiated performance or correctness issue remained after the final candidate audit.

Review status: static review converged on this frozen three-comment addition plus the four existing live threads.

* Whether this write can emit data files. A delete-only MERGE returns false so a connector may
* allow position-delete output even when the table has read-only column types.
*/
default boolean isWritesDataFiles() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Bump the connector plugin API for these new SPI methods

This method and ConnectorScanPlanProvider.canServeMetadataOnlyCount() extend the public connector SPI, but the PR leaves the kernel/plugin API at 3.0. A new Iceberg plugin is therefore admitted by an old 3.0 FE; because connector SPI classes are parent-first, planWrite() then invokes isWritesDataFiles() on the old kernel interface and fails with NoSuchMethodError. Please apply the repository's required major bump (including the test pin/baseline), and include these reachable provider/handle types in the frozen surface so this cannot evade the guard.

ConnectorScanPlanProvider scanProvider = resolveScanProvider();
if (isTableLevelCountStarPushdown() && conjuncts.isEmpty() && scanProvider != null) {
metadataCountProven = onPluginClassLoader(scanProvider,
() -> scanProvider.canServeMetadataOnlyCount(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Prove metadata COUNT on the same pinned handle that is planned

This capability check runs before pinMvccSnapshot() updates currentHandle, while planScan() later uses the pinned handle and recomputes whether the summary can serve the count. For a time-travel/reference query (or a snapshot change between phases), the early latest snapshot can return true, disabling the old-BE fence, but the selected snapshot can contain deletes or missing counters and fall back to real Variant file ranges. Please apply the statement pin before using this escape hatch, or base the bypass on the actual planned ranges; add a test where latest is countable but the pinned snapshot is not.

return;
}
for (Backend backend : backends) {
if (backend.isSmoothUpgradeSrc()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Fence ordinary rolling-upgrade backends too

isSmoothUpgradeSrc() identifies only the cloud colocated smooth-upgrade source: its production setter is in CloudSystemInfoService, while ordinary backends default this bit to false. During a normal mixed-version cluster upgrade, an old BE therefore passes this loop and can be assigned an Iceberg Variant scan even though it lacks the new carrier/native reader. Please gate on an explicit BE capability or execution version that covers every eligible backend, and test an old non-cloud backend rather than manually setting the cloud-only flag.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 71.26% (2120/2975) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.40% (25861/43535)
Line Coverage 43.52% (260624/598842)
Region Coverage 39.24% (206818/527073)
Branch Coverage 40.59% (94725/233369)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 37.04% (30/81) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 73.10% (2174/2974) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.56% (31653/42454)
Line Coverage 58.77% (349529/594779)
Region Coverage 54.94% (290517/528811)
Branch Coverage 55.98% (130645/233383)

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Aug 4, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66413

Problem Summary: Review follow-ups found that Variant metadata pruning could bypass an earlier error-producing predicate, late runtime-filter refresh could shift deferred Variant output slots, and mixed-version clusters could schedule unsupported Variant scans or delete-only MERGE plans. The connector SPI version also did not reflect its expanded public surface. In addition, debug Boolean validation filtered and copied large nullable complex columns even when they contained no Boolean values, which could exhaust query memory. This change preserves the safe pruning prefix and scan layout, introduces execution-version compatibility gates, bumps and freezes the connector SPI surface, pins metadata-count checks to the selected snapshot, and skips allocation-heavy Boolean filtering when no Boolean subcolumn exists.

### Release note

Iceberg Variant scans and delete-only MERGE now enforce rolling-upgrade compatibility, and debug column validation avoids copying non-Boolean complex payloads.

### Check List (For Author)

- Test: Unit Test and Regression Test
    - Focused ASAN BE unit tests for Boolean validation, Variant scan refresh, metadata pruning, page filtering, and MERGE compatibility
    - FE compatibility and connector SPI surface unit tests
    - Generated Iceberg Variant regression golden output
    - FE Checkstyle and BE clang-format
- Behavior changed: Yes. Unsafe metadata pruning and mixed-version Variant execution are rejected or conservatively evaluated, and non-Boolean nullable complex columns are validated without copying their payload.
- Does this need documentation: No
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66446

Problem Summary: Remove the forward port of projected shredded Variant state preservation and its master-only callback adaptation so the branch returns exactly to the state before that pull request.

### Release note

None

### Check List (For Author)

- Test: No need to test (the resulting tree exactly matches the pre-apache#66446 state)
- Behavior changed: Yes (removes the apache#66446 forward port)
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66412

Problem Summary: Forward-port compound Parquet Page Index pruning to master. AND nodes intersect conservative candidate ranges, OR nodes union them, and unavailable OR branches retain the complete range. The master adaptation also keeps compound predicates behind the existing metadata-pruning safety fence.

### Release note

Support compound Parquet Page Index pruning in File Scanner V2.

### Check List (For Author)

- Test: Unit Test (2 focused BE ASAN tests passed)
- Behavior changed: Yes (enables conservative compound Page Index pruning)
- Does this need documentation: Yes (included in this commit)
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66413

Problem Summary: The forward-ported Iceberg case expected the branch-4.1 nested-path rendering, while plugin-driven scans on master render projected paths by logical column name. The Paimon DESC golden also omitted the Extra column that master returns for every schema row. Align both expectations with the observed master output without weakening the execution checks.

### Release note

None

### Check List (For Author)

- Test: Regression framework unit tests (3 tests passed) and static validation of the six-column DESC golden shape.

- Behavior changed: No

- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29326 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e8e8c6b6ab08a1b6b5cb83266086d95fef206e4, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17741	3993	3971	3971
q2	2002	330	198	198
q3	10310	1409	835	835
q4	4683	473	340	340
q5	7491	847	564	564
q6	176	166	133	133
q7	739	800	599	599
q8	9335	1669	1641	1641
q9	5440	4115	4041	4041
q10	6768	1607	1376	1376
q11	520	371	341	341
q12	714	592	454	454
q13	18077	3295	2737	2737
q14	261	256	238	238
q15	q16	758	745	669	669
q17	1017	1013	901	901
q18	6637	5632	5558	5558
q19	1175	1204	1127	1127
q20	809	683	548	548
q21	6272	2832	2737	2737
q22	460	386	318	318
Total cold run time: 101385 ms
Total hot run time: 29326 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4948	4677	4596	4596
q2	302	330	209	209
q3	5375	5260	4633	4633
q4	2184	2252	1430	1430
q5	4509	4634	4403	4403
q6	231	173	126	126
q7	1875	1672	1477	1477
q8	2359	2030	2033	2030
q9	7234	7153	6720	6720
q10	4229	4238	3788	3788
q11	517	375	351	351
q12	698	716	497	497
q13	2938	3237	2726	2726
q14	282	277	256	256
q15	q16	658	690	603	603
q17	1243	1231	1225	1225
q18	12114	11054	11882	11054
q19	1119	1059	1102	1059
q20	2198	2203	1929	1929
q21	5290	4582	4651	4582
q22	518	473	400	400
Total cold run time: 60821 ms
Total hot run time: 54094 ms

@Gabriel39
Gabriel39 force-pushed the dev/forward-port-66397-66302-master branch from 1e8e8c6 to a1de39a Compare August 6, 2026 14:47
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 166373 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1e8e8c6b6ab08a1b6b5cb83266086d95fef206e4, data reload: false

query5	4327	590	475	475
query6	472	221	215	215
query7	4865	563	337	337
query8	318	157	146	146
query9	8815	4111	4068	4068
query10	459	355	299	299
query11	5892	2203	2049	2049
query12	154	103	99	99
query13	1270	632	457	457
query14	6073	4337	4032	4032
query14_1	3848	3851	3826	3826
query15	208	198	172	172
query16	1009	473	443	443
query17	953	732	523	523
query18	2410	467	332	332
query19	206	186	142	142
query20	102	106	102	102
query21	243	157	132	132
query22	13068	12999	12761	12761
query23	15667	15157	14680	14680
query23_1	14772	14768	14582	14582
query24	7745	1738	1229	1229
query24_1	1262	1236	1258	1236
query25	547	418	353	353
query26	1344	353	222	222
query27	2601	618	384	384
query28	4603	2074	2043	2043
query29	1069	608	468	468
query30	337	267	226	226
query31	1204	1116	1047	1047
query32	101	59	60	59
query33	519	309	245	245
query34	1210	1099	659	659
query35	740	760	633	633
query36	798	803	716	716
query37	161	106	92	92
query38	1824	1791	1683	1683
query39	832	826	767	767
query39_1	797	784	791	784
query40	262	163	145	145
query41	64	65	65	65
query42	96	94	103	94
query43	333	333	278	278
query44	1500	782	778	778
query45	189	183	169	169
query46	1043	1210	696	696
query47	1526	1539	1437	1437
query48	403	429	316	316
query49	603	418	304	304
query50	1081	427	339	339
query51	10683	10699	10561	10561
query52	89	95	76	76
query53	266	284	192	192
query54	295	238	234	234
query55	76	77	67	67
query56	315	294	300	294
query57	1018	1006	921	921
query58	285	259	267	259
query59	1566	1658	1379	1379
query60	321	288	271	271
query61	182	177	181	177
query62	403	330	273	273
query63	241	205	203	203
query64	3048	1170	1012	1012
query65	3906	3809	3811	3809
query66	1832	503	378	378
query67	28240	28012	27390	27390
query68	3357	1532	1061	1061
query69	431	315	270	270
query70	928	817	809	809
query71	374	351	323	323
query72	3273	2266	2373	2266
query73	801	740	458	458
query74	4646	4483	4284	4284
query75	2372	2354	2000	2000
query76	2438	1145	797	797
query77	358	375	279	279
query78	11125	11139	10712	10712
query79	1385	1156	750	750
query80	1236	559	463	463
query81	544	323	293	293
query82	649	178	137	137
query83	373	328	297	297
query84	331	153	130	130
query85	990	613	514	514
query86	407	240	228	228
query87	1983	1950	1839	1839
query88	3755	2840	2816	2816
query89	393	319	290	290
query90	1884	209	192	192
query91	208	188	169	169
query92	66	65	55	55
query93	1719	1557	1064	1064
query94	704	351	325	325
query95	789	616	482	482
query96	1088	824	360	360
query97	2441	2448	2361	2361
query98	198	189	185	185
query99	729	745	613	613
Total cold run time: 254773 ms
Total hot run time: 166373 ms

@github-actions

github-actions Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31112597055

Please trigger /review again after that time.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29018 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1de39a3232393b868b8ebb4e8bd839ec0889c33, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17656	3934	3927	3927
q2	1996	312	204	204
q3	10307	1454	824	824
q4	4682	479	351	351
q5	7542	860	549	549
q6	185	171	135	135
q7	771	806	598	598
q8	9375	1698	1555	1555
q9	5940	4133	4098	4098
q10	6783	1643	1377	1377
q11	511	350	321	321
q12	736	574	451	451
q13	18090	3332	2727	2727
q14	263	263	250	250
q15	q16	747	731	661	661
q17	1077	986	998	986
q18	6580	5642	5620	5620
q19	1163	1231	1095	1095
q20	783	659	570	570
q21	5571	2575	2416	2416
q22	437	349	303	303
Total cold run time: 101195 ms
Total hot run time: 29018 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4295	4212	4214	4212
q2	272	322	213	213
q3	4579	4998	4355	4355
q4	2154	2261	1433	1433
q5	4232	4106	4122	4106
q6	226	173	130	130
q7	1736	1572	1392	1392
q8	2227	1947	1884	1884
q9	6825	6744	6722	6722
q10	4267	4239	3806	3806
q11	521	387	359	359
q12	726	723	516	516
q13	2937	3253	2764	2764
q14	291	295	256	256
q15	q16	682	711	612	612
q17	1247	1229	1236	1229
q18	12088	11023	11863	11023
q19	1095	1087	1074	1074
q20	2220	2190	1924	1924
q21	5304	4593	4531	4531
q22	530	478	415	415
Total cold run time: 58454 ms
Total hot run time: 52956 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 166512 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1de39a3232393b868b8ebb4e8bd839ec0889c33, data reload: false

query5	4318	594	455	455
query6	463	219	211	211
query7	4864	620	350	350
query8	338	163	149	149
query9	8773	3986	4064	3986
query10	455	346	291	291
query11	5866	2199	2036	2036
query12	152	95	94	94
query13	1234	569	438	438
query14	6049	4240	3950	3950
query14_1	3787	3736	3742	3736
query15	194	195	175	175
query16	982	482	450	450
query17	927	701	532	532
query18	2446	467	344	344
query19	214	189	152	152
query20	106	100	102	100
query21	237	165	138	138
query22	13094	12878	12762	12762
query23	15908	14958	14472	14472
query23_1	14784	14692	14671	14671
query24	7442	1707	1240	1240
query24_1	1247	1251	1264	1251
query25	555	449	378	378
query26	1334	358	212	212
query27	2625	585	385	385
query28	4580	2064	2021	2021
query29	1093	626	491	491
query30	350	265	226	226
query31	1160	1115	1069	1069
query32	114	64	61	61
query33	534	324	248	248
query34	1245	1145	634	634
query35	727	759	642	642
query36	771	773	718	718
query37	156	115	93	93
query38	1821	1776	1665	1665
query39	818	817	837	817
query39_1	813	788	767	767
query40	246	178	144	144
query41	71	71	69	69
query42	97	97	96	96
query43	319	322	277	277
query44	1479	788	782	782
query45	197	175	173	173
query46	1053	1130	726	726
query47	1530	1535	1469	1469
query48	427	412	301	301
query49	590	425	311	311
query50	1090	452	345	345
query51	10524	10813	10474	10474
query52	90	89	79	79
query53	275	277	209	209
query54	300	252	233	233
query55	77	73	73	73
query56	308	315	316	315
query57	1029	1003	918	918
query58	285	269	272	269
query59	1547	1586	1373	1373
query60	339	314	268	268
query61	173	169	184	169
query62	391	318	266	266
query63	238	195	201	195
query64	2846	1015	837	837
query65	3902	3827	3809	3809
query66	1817	480	358	358
query67	28221	28116	28041	28041
query68	3459	1571	971	971
query69	425	300	272	272
query70	880	777	779	777
query71	389	342	321	321
query72	3260	2668	2293	2293
query73	837	736	400	400
query74	4736	4525	4286	4286
query75	2373	2331	2015	2015
query76	2439	1142	785	785
query77	342	373	279	279
query78	11193	11071	10573	10573
query79	1345	1140	778	778
query80	672	557	480	480
query81	454	332	292	292
query82	632	178	141	141
query83	398	328	291	291
query84	333	165	135	135
query85	922	599	518	518
query86	323	235	219	219
query87	1957	1936	1861	1861
query88	3813	2829	2822	2822
query89	403	316	291	291
query90	1944	200	204	200
query91	203	192	164	164
query92	61	62	57	57
query93	1781	1551	994	994
query94	559	351	316	316
query95	796	572	490	490
query96	1050	774	340	340
query97	2474	2461	2335	2335
query98	201	187	188	187
query99	719	729	636	636
Total cold run time: 253464 ms
Total hot run time: 166512 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1de39a3232393b868b8ebb4e8bd839ec0889c33, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.25	0.14	0.13
query4	1.61	0.14	0.14
query5	0.24	0.22	0.22
query6	1.16	0.80	0.81
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.37	0.31	0.30
query10	0.54	0.55	0.56
query11	0.19	0.14	0.14
query12	0.18	0.14	0.14
query13	0.46	0.47	0.46
query14	1.00	0.99	0.99
query15	0.60	0.59	0.58
query16	0.32	0.31	0.32
query17	1.09	1.08	1.12
query18	0.22	0.19	0.20
query19	2.02	1.97	1.95
query20	0.02	0.01	0.01
query21	15.45	0.21	0.14
query22	4.92	0.06	0.06
query23	16.13	0.31	0.12
query24	2.90	0.44	0.33
query25	0.10	0.06	0.04
query26	0.74	0.19	0.14
query27	0.05	0.04	0.03
query28	3.58	0.78	0.35
query29	12.48	4.02	3.22
query30	0.27	0.16	0.15
query31	2.76	0.55	0.32
query32	3.22	0.58	0.49
query33	3.20	3.15	3.16
query34	15.52	3.91	3.28
query35	3.27	3.20	3.23
query36	0.55	0.42	0.42
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.17	0.16	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.04
Total cold run time: 96.11 s
Total hot run time: 23.85 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 77.74% (3146/4047) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.93% (26262/43821)
Line Coverage 44.35% (267775/603813)
Region Coverage 40.11% (213393/531991)
Branch Coverage 41.57% (98207/236248)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 80.69% (3259/4039) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.98% (32468/42730)
Line Coverage 60.88% (365116/599719)
Region Coverage 57.66% (307687/533669)
Branch Coverage 59.02% (139418/236230)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 20.88% (19/91) 🎉
Increment coverage report
Complete coverage report

…pache#66529)

## Proposed changes

- make Iceberg compatibility gates conservative and bounded while
pinning column handles to historical schemas
- reset connector metadata across INSERT retries and align V1/V2
defaults, required-field checks, and position-delete row projection
- support primitive partition sources nested in structs, including
nullable-parent propagation and regression coverage
- cache the compact equality-delete field-ID projection by immutable
table snapshot even when the optional full manifest cache is disabled
- resolve nested partition sources through the top-level Nereids slot ID
and fail closed when stable Iceberg IDs are unavailable

## Compatibility-gate trade-offs

- Equality-delete fencing remains conservative across all delete
manifests in the selected snapshot, including partition-pruned scans.
The snapshot-scoped field-ID cache removes repeated manifest walks
without weakening correctness; initial-load failures still fail closed
and remain retryable.
- Requiredness fencing intentionally uses bounded schema-history
inspection rather than an O(snapshot-count) ancestry walk because
snapshot schema IDs are optional. Once a projected requiredness hazard
exists, every non-empty selected snapshot is fenced. This can reduce
rolling-upgrade availability but cannot create a correctness false
negative.

## Testing

- `mvn -pl fe-core,fe-connector/fe-connector-iceberg -am
-Dtest=PhysicalExternalRowLevelMergeSinkTest,IcebergManifestCacheTest,IcebergScanPlanProviderTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `mvn -pl fe-connector/fe-connector-iceberg -am
-Dtest=IcebergScanPlanProviderTest,IcebergConnectorMetadataTest,IcebergWritePlanProviderTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `mvn -pl fe-core -am
-Dtest=ConnectorStatementScopeTest,InsertIntoTableCommandTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `./run-be-ut.sh --run
--filter=SchemaTest.*:VIcebergTableWriterTest.*:IcebergReaderTest.v1_materializes_non_finite_initial_defaults:IcebergV2ReaderTest.PreparesIcebergNonFiniteInitialDefaults:IcebergPositionDeleteSysTableV2ProfileTest.*`
- FE Checkstyle for all affected modules
- clang-format 16 check for all changed C/C++ files
…#66470)

## Summary

- backport the floating-point pruning correctness fixes to branch-4.1
for File Scanner V2
- preserve Doris NaN and signed-zero equality semantics in V2 Parquet
Bloom, min/max, and IN-predicate pruning
- keep pruning conservative when Parquet statistics omit NaNs or a Bloom
encoding cannot safely prove absence
- leave the legacy File Scanner V1 path unchanged

## Testing

- clang-format 16 check on all affected C/C++ files
- compiled every affected production and test translation unit after
rebasing onto branch-4.1
- 121 focused BE unit tests passed, including expression, hybrid-set,
and native V2 Parquet pruning coverage
…che#66471)

## Summary

- backport nested Parquet Bloom-filter pruning support to branch-4.1
- resolve struct and list leaf predicates for equality, null-safe
equality, and IN probes
- retain table-level evaluation when schema mapping, nullability, or
expression localization makes early filtering unsafe
- preserve filter order and merge deferred complex projections so
rejected localization cannot bypass validation or drop residual-filter
children
- keep nested Variant leaf predicates eager while validating nullability
only at mapped table-schema levels
- add pruning diagnostics and focused coverage for Parquet, ORC, column
mapping, and TableReader paths

## Testing

- clang-format 16 check on all affected C/C++ files
- 162 focused BE unit tests across ExprZonemapFilterTest,
ColumnMapperScanRequestTest, ParquetBloomFilterPruningTest, and
TableReaderTest
- 38 focused mapper tests, including
ColumnMapperTest.NestedVariantAllAccessPathKeepsPhysicalTypedLeaf from
BE UT build 1016464
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31172050393

Please trigger /review again after that time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants